[00:01.020 --> 00:04.030]  Hello and welcome to 10,000 Dependencies Under the Sea,
[00:04.030 --> 00:07.090]  Exploring and Securing Open Source Dependencies.
[00:07.380 --> 00:08.830]  My name is Greg Horton.
[00:08.830 --> 00:11.210]  I'm a Product Security Engineer at Slack.
[00:11.210 --> 00:13.990]  And I'm Ryan Slama, an Associate Software Engineer
[00:13.990 --> 00:17.050]  on the Product Security Foundations team.
[00:17.690 --> 00:21.490]  At Slack, AppSec is product security.
[00:21.490 --> 00:23.710]  Our product organization is split
[00:23.710 --> 00:26.010]  into Classics and Foundations.
[00:26.010 --> 00:27.650]  Classics were the original team,
[00:27.650 --> 00:29.690]  and they focus on most of the traditional
[00:29.690 --> 00:32.970]  AppSec responsibilities, like reviewing new features,
[00:32.970 --> 00:34.630]  penetration testing, and running
[00:34.830 --> 00:36.890]  a healthy bug bounty program.
[00:37.270 --> 00:38.910]  I'm on the Foundations team,
[00:38.910 --> 00:41.890]  which focuses on reducing risk through automated tooling
[00:41.890 --> 00:45.250]  and creating secure by default libraries and patterns.
[00:45.410 --> 00:47.010]  Both of these teams work together
[00:47.010 --> 00:49.790]  to ensure the security of Slack, the product,
[00:49.790 --> 00:52.070]  using a multifaceted approach.
[00:53.330 --> 00:56.070]  For some context, what is Slack?
[00:56.070 --> 00:58.890]  Slack is a channel-based messaging platform.
[00:58.890 --> 01:02.390]  And with Slack, people can work together more effectively,
[01:02.390 --> 01:04.350]  connect all their tools and services,
[01:04.350 --> 01:07.270]  and find the information they need to do their best work,
[01:07.270 --> 01:10.470]  all within a secure, enterprise-grade environment.
[01:11.070 --> 01:14.750]  Let's begin by talking a bit about the Slack stack.
[01:15.270 --> 01:17.690]  Users expect Slack to work everywhere.
[01:17.690 --> 01:20.250]  To that end, we use and are a partial maintainer
[01:20.250 --> 01:21.450]  of the Electron project
[01:21.450 --> 01:24.670]  for a consistent experience across devices.
[01:25.010 --> 01:27.210]  In practice, this means our front-end web code
[01:27.210 --> 01:30.350]  also runs in all of our desktop clients.
[01:30.590 --> 01:33.210]  Our back-end is primarily in Hacklang,
[01:33.210 --> 01:34.850]  Facebook's fork of PHP,
[01:34.850 --> 01:37.190]  with strong typing and other enhancements.
[01:37.350 --> 01:39.950]  We also have some services written in Go,
[01:39.950 --> 01:41.410]  like our caching layer.
[01:41.470 --> 01:45.010]  Finally, our mobile apps are primarily in Swift and Kotlin,
[01:45.010 --> 01:47.610]  but we won't be talking about them much today.
[01:49.070 --> 01:52.710]  Today, we'll be talking about an OWASP top 10 issue,
[01:52.710 --> 01:55.030]  using components with known vulnerabilities.
[01:55.030 --> 01:58.430]  Specifically, we'll be focusing on vulnerability management
[01:58.430 --> 02:00.470]  for third-party dependencies.
[02:01.850 --> 02:05.410]  Our story today begins with an intern project.
[02:05.470 --> 02:06.630]  Matt Dwanzig and I,
[02:06.630 --> 02:08.750]  who are now full-time engineers at Slack,
[02:08.750 --> 02:11.350]  were interning on the Product Security Foundations team
[02:11.350 --> 02:12.650]  last summer.
[02:12.810 --> 02:15.450]  When we got there, we were given an open-ended project
[02:15.450 --> 02:18.610]  of understanding and limiting our dependency risk.
[02:18.610 --> 02:21.150]  Today, we will walk you through our journey
[02:21.150 --> 02:22.570]  building a tool,
[02:22.570 --> 02:26.530]  and Greg will walk you through how we implemented the tool
[02:26.530 --> 02:28.010]  and built a process around it
[02:28.010 --> 02:30.930]  to actually limit the risk at Slack.
[02:32.170 --> 02:36.370]  Modern codebases often require tons of third-party code.
[02:36.490 --> 02:38.950]  At Slack, we have one main repository
[02:38.950 --> 02:42.950]  that contains our entire front-end and most of our back-end.
[02:43.250 --> 02:48.430]  Our main repository currently requires over 6,500 packages.
[02:48.450 --> 02:50.610]  A year ago, we had half that.
[02:50.810 --> 02:52.110]  This trend would be concerning
[02:52.110 --> 02:55.150]  if we didn't have systems in place to limit risk.
[02:55.270 --> 02:58.010]  All of our first-party code has to be reviewed,
[02:58.010 --> 03:00.470]  but what about random stuff you find on GitHub?
[03:00.470 --> 03:01.710]  That's exempt.
[03:01.930 --> 03:05.230]  Clearly, some process must be in place to manage risk.
[03:07.050 --> 03:10.130]  It's important to note that we have a mature process
[03:10.130 --> 03:14.510]  for adding packages, and the count still doubled in a year.
[03:14.890 --> 03:16.830]  For PRs that add packages,
[03:16.830 --> 03:19.350]  developers have to explain why it's needed.
[03:20.070 --> 03:22.170]  Packages must be actively maintained
[03:22.170 --> 03:24.170]  and save meaningful engineering effort
[03:24.170 --> 03:26.790]  over just building the functionality ourselves.
[03:27.230 --> 03:29.770]  Additionally, all packages are assigned a team
[03:29.770 --> 03:31.750]  or a directly responsible individual
[03:31.750 --> 03:34.610]  to update them and maintain them.
[03:34.610 --> 03:38.210]  This is especially important for security updates and fixes.
[03:38.870 --> 03:41.890]  So how did our package count double?
[03:42.550 --> 03:44.830]  In a word, NPM.
[03:44.830 --> 03:49.990]  We only directly require 350 packages in our package JSON.
[03:50.010 --> 03:53.110]  However, when you resolve the nested dependencies,
[03:53.110 --> 03:54.830]  our dependency tree expands
[03:54.830 --> 03:58.610]  to almost 6,500 unique versions of packages.
[03:59.970 --> 04:03.690]  Running this much third-party code can be a risk.
[04:03.690 --> 04:05.490]  The downside of using common software
[04:05.490 --> 04:08.030]  is that we often find out about vulnerabilities
[04:08.030 --> 04:10.670]  at the same time as everyone else.
[04:10.670 --> 04:13.810]  Let's take a look at some examples of related issues.
[04:14.890 --> 04:17.990]  One of the most notable was Equifax.
[04:17.990 --> 04:19.290]  Equifax got hacked
[04:19.290 --> 04:22.650]  because they were running an old version of Apache struts
[04:22.650 --> 04:24.690]  that had known vulnerabilities.
[04:24.850 --> 04:29.910]  This hack leaked 143 million social security numbers.
[04:30.050 --> 04:31.890]  Preventing this could have been as simple
[04:31.890 --> 04:34.110]  as upgrading their version of Apache struts
[04:34.110 --> 04:37.830]  to one that did not have publicly known vulnerabilities.
[04:40.150 --> 04:42.150]  Vulnerabilities are also discovered
[04:42.150 --> 04:45.330]  in old versions of popular NPM packages.
[04:45.690 --> 04:49.230]  For example, both of these advisories were issued this year.
[04:49.410 --> 04:54.150]  Next.js and AngularJS are respected, well-maintained packages.
[04:54.150 --> 04:57.530]  But running those in your production codebases
[04:57.530 --> 05:02.210]  without adequately maintaining them is a recipe for disaster
[05:02.210 --> 05:05.310]  because new vulnerabilities can be discovered at any time
[05:05.310 --> 05:08.810]  no matter how many years you've been running the same code.
[05:09.770 --> 05:14.690]  NPM also had issues with malicious code in popular packages.
[05:14.690 --> 05:21.130]  For example, EventStream, which was a popular NPM library,
[05:21.130 --> 05:24.610]  required a dependency that had been compromised.
[05:24.610 --> 05:26.670]  And that compromised dependency included
[05:27.030 --> 05:29.370]  a targeted Bitcoin wallet stealer.
[05:29.370 --> 05:33.850]  And that malicious code was downloaded over 8 million times.
[05:35.570 --> 05:39.010]  ESLint Scope is an even more popular NPM package
[05:39.010 --> 05:41.690]  used for linting JavaScript code.
[05:41.690 --> 05:43.730]  And a compromised version was uploaded
[05:43.730 --> 05:46.450]  that exfiltrated NPM RC files,
[05:46.450 --> 05:49.050]  which contained package publishing credentials.
[05:49.050 --> 05:51.130]  Those credentials could allow an attacker
[05:51.130 --> 05:53.610]  to push a new version of a package
[05:53.610 --> 05:56.430]  as a semver patch with malicious code.
[05:56.430 --> 06:00.130]  And because of the way NPM's semver works,
[06:00.130 --> 06:01.810]  all installations of that package
[06:01.810 --> 06:03.830]  would automatically upgrade to the malicious code.
[06:05.830 --> 06:09.990]  And here's for some figures that might be slightly biased.
[06:09.990 --> 06:13.610]  Snyk found an 88% increase in library vulnerabilities
[06:13.610 --> 06:15.470]  over the past two years.
[06:15.470 --> 06:18.530]  And 78% of vulnerabilities that they found
[06:18.530 --> 06:20.950]  are in indirect dependencies.
[06:21.050 --> 06:23.170]  Just take a look at our dependency tree
[06:23.170 --> 06:25.210]  from earlier to understand why.
[06:25.210 --> 06:27.530]  When we only require 345,
[06:27.530 --> 06:30.830]  but end up with almost 6,500 packages,
[06:30.830 --> 06:33.150]  there's a lot more code beneath the surface
[06:33.150 --> 06:34.790]  than on top of it.
[06:35.750 --> 06:38.350]  And we're not the only ones who have this problem.
[06:38.490 --> 06:39.990]  Our customers know about this problem
[06:39.990 --> 06:41.490]  and they ask us about it.
[06:41.490 --> 06:43.210]  We need a good program in place
[06:43.210 --> 06:45.850]  so we can show them that we take care of their data
[06:45.850 --> 06:47.550]  and we limit risk.
[06:48.430 --> 06:50.650]  What do we do about this problem now?
[06:51.410 --> 06:53.670]  When we started, we defined three goals
[06:53.670 --> 06:54.970]  for a good solution.
[06:55.070 --> 06:57.370]  The first is we want to detect vulnerabilities
[06:57.370 --> 06:59.250]  as soon as they're publicly available
[06:59.250 --> 07:01.550]  because we want to be on top of the fixes
[07:01.550 --> 07:03.130]  as quickly as possible.
[07:03.710 --> 07:05.970]  Second, we want to track any vulnerabilities
[07:05.970 --> 07:08.090]  or weaknesses in our code base.
[07:08.090 --> 07:10.290]  We want to be able to have insight
[07:10.290 --> 07:12.410]  into where we have risk
[07:12.410 --> 07:14.770]  and so we can better fix it.
[07:14.950 --> 07:17.850]  Finally, we want to alert when we have a problem.
[07:17.870 --> 07:19.770]  No developer should ever have to think,
[07:19.770 --> 07:21.270]  oh, I should scan my repo
[07:21.270 --> 07:23.270]  before I deployed a production.
[07:23.670 --> 07:25.930]  We want to alert when we have vulnerabilities
[07:25.930 --> 07:28.670]  so this just happens automatically.
[07:29.490 --> 07:31.470]  The next question we asked was,
[07:31.470 --> 07:33.130]  can we use an off-the-shelf tool?
[07:33.150 --> 07:35.430]  But our requirements ended up making
[07:35.430 --> 07:37.070]  that surprisingly difficult.
[07:37.710 --> 07:39.890]  First of all, we use Hacklang,
[07:39.890 --> 07:42.750]  which is not a popular language outside Facebook,
[07:42.750 --> 07:44.390]  so it was difficult to find vendors
[07:44.390 --> 07:45.810]  that supported it.
[07:45.870 --> 07:47.830]  Second, we use GitHub Enterprise
[07:47.830 --> 07:49.670]  rather than GitHub Cloud,
[07:49.670 --> 07:52.210]  which makes it harder for Dependabot
[07:52.210 --> 07:54.850]  because Dependabot still isn't out
[07:54.850 --> 07:55.670]  for GitHub Enterprise
[07:55.670 --> 07:58.470]  and it was even further from being out
[07:58.790 --> 08:01.090]  a year ago when we began this journey.
[08:02.170 --> 08:04.510]  Next, we needed a tool that will scan
[08:04.510 --> 08:06.610]  the entire dependency trees.
[08:06.930 --> 08:08.170]  Knowing about vulnerabilities
[08:08.170 --> 08:10.010]  and packages we require directly
[08:10.010 --> 08:11.170]  isn't enough.
[08:11.170 --> 08:12.370]  As we saw earlier,
[08:12.370 --> 08:13.650]  there are so many more packages
[08:13.650 --> 08:14.530]  beneath the surface
[08:14.530 --> 08:15.630]  and we need to make sure
[08:15.630 --> 08:17.510]  that those are a stable foundation
[08:17.510 --> 08:19.450]  that we can build our app on.
[08:19.690 --> 08:21.430]  And finally, at Slack,
[08:21.430 --> 08:23.250]  we're heavy users of Slack
[08:23.250 --> 08:24.850]  as a product.
[08:24.850 --> 08:27.290]  All of our alerts and things like that
[08:27.290 --> 08:28.830]  just go in Slack.
[08:28.830 --> 08:31.190]  So we need a tool that has support
[08:31.190 --> 08:32.690]  for routing alerts
[08:32.690 --> 08:33.790]  for different code bases
[08:33.790 --> 08:35.910]  or packages to different teams.
[08:36.370 --> 08:38.370]  And finally, you might be saying,
[08:38.370 --> 08:39.350]  oh, we're a vendor
[08:39.350 --> 08:41.350]  and we have a tool that does all that.
[08:41.390 --> 08:43.910]  But we're still a relatively small company
[08:43.910 --> 08:45.030]  and we need something
[08:45.030 --> 08:46.330]  that's not going to cost us
[08:46.330 --> 08:47.230]  millions of dollars
[08:47.230 --> 08:49.030]  over the next few years.
[08:50.750 --> 08:52.690]  So we built Ossify.
[08:53.830 --> 08:55.830]  Our solutions to the three points
[08:55.830 --> 08:57.490]  mentioned earlier were first,
[08:57.490 --> 08:58.710]  the in-detection.
[08:58.710 --> 09:01.430]  We run daily scans of our code bases
[09:01.430 --> 09:04.410]  to both figure out
[09:04.410 --> 09:06.610]  what packages we're requiring.
[09:06.610 --> 09:08.870]  And also, we upload those packages
[09:08.870 --> 09:10.950]  to the Sonatype open source
[09:10.950 --> 09:12.090]  vulnerability index
[09:12.090 --> 09:13.870]  and see if there are
[09:13.870 --> 09:15.150]  any new vulnerabilities
[09:15.150 --> 09:16.190]  that have been reported
[09:16.190 --> 09:17.430]  for our packages.
[09:17.770 --> 09:19.270]  Second, we built a dashboard
[09:19.270 --> 09:21.850]  to track the status of repositories,
[09:21.850 --> 09:24.270]  as well as tracking remediation efforts
[09:24.270 --> 09:26.350]  for individual findings.
[09:26.350 --> 09:27.970]  And finally, we built
[09:27.970 --> 09:29.790]  robust Slack alerting,
[09:29.790 --> 09:32.330]  which will be covered more later.
[09:35.010 --> 09:37.830]  Ossify supports three package ecosystems.
[09:37.830 --> 09:39.630]  The first, of course, is Hackline,
[09:39.630 --> 09:40.870]  because that's our backend
[09:40.870 --> 09:42.210]  and that's where some of the most
[09:42.210 --> 09:44.330]  dangerous possibilities are.
[09:44.790 --> 09:46.910]  We added custom metadata
[09:46.910 --> 09:50.030]  to track composer packages upstream
[09:50.030 --> 09:52.170]  because we had to manually fork
[09:52.170 --> 09:53.850]  and vendor some of these packages
[09:53.850 --> 09:55.250]  to add strong types
[09:55.250 --> 09:57.510]  and other Hackline features.
[09:57.910 --> 09:59.950]  Second, we added NPM,
[09:59.950 --> 10:02.170]  because NPM is the biggest offender
[10:02.170 --> 10:04.790]  for vast amounts of packages.
[10:04.790 --> 10:06.270]  And we found, by far,
[10:06.270 --> 10:07.250]  the most findings
[10:07.250 --> 10:09.530]  coming from NPM packages.
[10:09.890 --> 10:12.130]  And finally, we added Go support
[10:12.130 --> 10:13.050]  because a number
[10:13.050 --> 10:14.910]  of our high value services,
[10:14.910 --> 10:15.930]  like our caching layer,
[10:15.930 --> 10:16.910]  are written in Go.
[10:18.530 --> 10:20.590]  Here is what the Ossify dashboard
[10:20.590 --> 10:21.490]  looks like.
[10:21.490 --> 10:23.090]  This is an example repository
[10:23.090 --> 10:24.530]  created for this demo
[10:24.530 --> 10:25.590]  just to show you
[10:25.590 --> 10:26.690]  what it looks like
[10:26.690 --> 10:28.530]  when you scan a repository.
[10:28.730 --> 10:29.890]  So we support scanning
[10:29.890 --> 10:31.130]  multiple branches,
[10:31.130 --> 10:32.690]  which could be used potentially
[10:32.690 --> 10:34.650]  for future CI integration.
[10:35.050 --> 10:36.050]  Here we have an example
[10:36.050 --> 10:37.390]  finding page.
[10:37.390 --> 10:39.090]  This is for NodeForge,
[10:39.090 --> 10:40.890]  which was being pulled in
[10:40.890 --> 10:43.630]  in our sample repo's package JSON.
[10:43.930 --> 10:45.350]  NodeForge has a weakness,
[10:45.350 --> 10:46.350]  but we don't actually
[10:46.350 --> 10:47.790]  require NodeForge.
[10:47.790 --> 10:49.710]  Instead, we require a Google Auth
[10:49.710 --> 10:52.010]  library all the way on the left.
[10:52.010 --> 10:53.450]  We built out a dependency
[10:53.450 --> 10:55.130]  graph tool because we want
[10:55.130 --> 10:56.490]  our developers to understand
[10:56.490 --> 10:57.470]  where the vulnerability
[10:57.470 --> 10:59.210]  is coming from and what packages
[10:59.210 --> 11:00.890]  need upgraded or removed
[11:00.890 --> 11:02.390]  to fix the issue.
[11:04.290 --> 11:05.610]  Here's a more complicated
[11:05.610 --> 11:07.070]  dependency graph.
[11:07.470 --> 11:10.210]  This is from a dev dependency.
[11:10.490 --> 11:12.210]  The weakness was on the right
[11:12.210 --> 11:13.510]  in isURL.
[11:13.510 --> 11:15.150]  However, if you trace back
[11:15.150 --> 11:16.550]  all the way to the left,
[11:16.550 --> 11:18.030]  it was actually some Gulp plugins
[11:18.030 --> 11:19.430]  we were using as part of a build
[11:19.430 --> 11:21.530]  process that were pulling in
[11:21.530 --> 11:23.950]  the weak version of isURL.
[11:24.810 --> 11:26.690]  And finally, like any good
[11:26.690 --> 11:27.890]  security tool, it comes with
[11:27.890 --> 11:28.810]  dark mode.
[11:28.810 --> 11:29.910]  Now I'm going to hand things
[11:29.910 --> 11:31.370]  off to Greg to cover everything
[11:31.370 --> 11:33.010]  that happened after the original
[11:33.010 --> 11:34.550]  development of the tool.
[11:38.070 --> 11:39.710]  Thanks, Ryan, for that great
[11:39.710 --> 11:40.930]  overview of the tool.
[11:41.110 --> 11:42.110]  So now that we have this
[11:42.110 --> 11:43.350]  fully featured tool that
[11:43.350 --> 11:45.070]  did everything that we wanted,
[11:45.070 --> 11:46.170]  it was time to integrate it
[11:46.170 --> 11:47.570]  into our wider processes
[11:47.570 --> 11:48.690]  here at Slack.
[11:49.170 --> 11:50.970]  So this would be easy, right?
[11:50.970 --> 11:53.790]  You know, we have an app that
[11:53.790 --> 11:55.830]  scans our repositories,
[11:55.830 --> 11:56.750]  looks at our third-party
[11:57.550 --> 11:59.330]  integrations, sees if there's
[11:59.330 --> 12:00.990]  any vulnerabilities, and then
[12:00.990 --> 12:02.390]  lets us know about them.
[12:02.890 --> 12:04.410]  So our first workflow for this
[12:04.410 --> 12:05.890]  tool was that Ossify would
[12:05.890 --> 12:09.390]  scan daily and then post to a
[12:09.390 --> 12:10.850]  Slack channel any currently
[12:10.850 --> 12:12.930]  known vulnerabilities.
[12:12.930 --> 12:15.210]  And using the power of Slack,
[12:15.210 --> 12:16.910]  we could make Slack notifications
[12:16.910 --> 12:19.410]  that were actionable,
[12:19.410 --> 12:20.970]  meaning that we could give the
[12:20.970 --> 12:21.750]  information that was needed
[12:21.750 --> 12:23.650]  for quick remediation.
[12:23.650 --> 12:25.050]  They were non-obtrusive, so we
[12:25.050 --> 12:26.190]  could set those notifications
[12:26.190 --> 12:28.030]  to snooze or ignore if they
[12:28.030 --> 12:29.550]  weren't relevant to us.
[12:29.710 --> 12:32.370]  And then make them configurable
[12:32.370 --> 12:34.350]  so we could notify individual
[12:34.350 --> 12:36.550]  users and specific channels
[12:37.290 --> 12:39.590]  about these dependencies.
[12:40.250 --> 12:41.690]  Here's an example of what
[12:41.690 --> 12:42.970]  those looked like.
[12:43.010 --> 12:44.190]  So as you can see, this
[12:44.190 --> 12:45.550]  dependency detection of the
[12:45.550 --> 12:48.670]  app, it scanned a repository,
[12:48.670 --> 12:49.810]  it found some vulnerabilities
[12:49.810 --> 12:52.130]  in some third-party libraries,
[12:52.810 --> 12:54.410]  and it would tell us about them.
[12:54.410 --> 12:56.190]  And as you can see, also,
[12:56.190 --> 12:57.890]  you could set them to ignore,
[12:57.890 --> 13:00.090]  you could snooze them, etc.
[13:01.010 --> 13:02.150]  But the problem with that is
[13:02.150 --> 13:04.170]  that this channel was way
[13:04.170 --> 13:05.170]  too noisy.
[13:05.290 --> 13:06.550]  Every single day, it was
[13:06.550 --> 13:07.970]  posting all the vulnerabilities
[13:07.970 --> 13:10.630]  that were found, and they
[13:10.630 --> 13:12.090]  haven't been remediated yet,
[13:12.090 --> 13:13.630]  so it would repeat a lot of
[13:13.630 --> 13:14.550]  the ones that we'd seen the
[13:14.550 --> 13:15.350]  day before.
[13:15.350 --> 13:16.470]  Maybe we didn't want to ignore
[13:16.470 --> 13:17.890]  them yet because they were
[13:17.890 --> 13:18.970]  still relevant or we were
[13:18.970 --> 13:19.770]  still trying to find somebody
[13:19.770 --> 13:20.810]  to fix them.
[13:21.010 --> 13:22.670]  But it was a very long list,
[13:22.670 --> 13:23.830]  and so we couldn't really
[13:24.610 --> 13:26.970]  work to work them down
[13:27.470 --> 13:28.590]  in this format.
[13:30.030 --> 13:31.730]  And also, blindly throwing
[13:31.730 --> 13:33.670]  them into a channel made it
[13:33.670 --> 13:35.310]  so it was everybody's problem,
[13:35.310 --> 13:36.670]  which in practice made it
[13:36.670 --> 13:37.750]  nobody's problem.
[13:37.750 --> 13:39.150]  Nobody took initiative to
[13:39.150 --> 13:41.250]  see which vulnerabilities
[13:41.250 --> 13:44.670]  were actually a threat to us,
[13:45.290 --> 13:47.050]  and nobody was taking
[13:47.050 --> 13:48.490]  responsibility to fix them.
[13:48.490 --> 13:49.610]  So it was pretty ineffective
[13:49.610 --> 13:50.890]  at solving our main problem
[13:50.890 --> 13:51.890]  at first, so we had to go
[13:51.890 --> 13:52.770]  back to the drawing board
[13:53.310 --> 13:54.890]  and figure that out.
[13:56.170 --> 13:57.250]  So we had to admit that
[13:57.250 --> 13:59.490]  Ossify, you know, was an
[13:59.490 --> 14:01.590]  effective tool, but it was
[14:01.590 --> 14:02.330]  not only reporting
[14:02.330 --> 14:03.690]  vulnerabilities, but it was
[14:03.690 --> 14:04.990]  also acting as an internal
[14:04.990 --> 14:06.230]  ticketing system.
[14:06.810 --> 14:07.690]  For something we wanted to
[14:07.690 --> 14:09.810]  maintain long term, we
[14:09.810 --> 14:11.030]  wanted Ossify to do one
[14:11.030 --> 14:12.070]  thing well, find
[14:12.070 --> 14:13.330]  vulnerabilities and report
[14:13.330 --> 14:16.190]  them. And to do this, we
[14:16.190 --> 14:17.190]  had some prior art that we
[14:17.190 --> 14:18.750]  could work from to figure
[14:18.750 --> 14:19.590]  out the best way to do that
[14:19.590 --> 14:20.410]  at Slack.
[14:21.210 --> 14:23.170]  And that answer was Jira.
[14:24.330 --> 14:26.330]  We have a Jira system, we
[14:26.330 --> 14:27.810]  have security tickets that
[14:27.810 --> 14:29.090]  go into our Jira that can
[14:29.090 --> 14:30.430]  be actioned upon by
[14:30.430 --> 14:33.030]  developers. And so our
[14:33.030 --> 14:34.730]  solution was to put it in
[14:34.730 --> 14:37.110]  Jira. At Slack, we have a
[14:37.110 --> 14:39.610]  method already for triaging
[14:39.610 --> 14:41.090]  security-based tickets that
[14:41.090 --> 14:42.610]  come from multiple sources,
[14:42.610 --> 14:44.190]  say if we have bug bounty
[14:44.190 --> 14:45.570]  reports or internal
[14:45.570 --> 14:47.750]  findings. Our SLAs, or
[14:47.750 --> 14:49.050]  service-level agreements,
[14:49.050 --> 14:50.770]  are 180 days for low
[14:50.770 --> 14:52.270]  tickets, 90 days for
[14:52.270 --> 14:53.870]  medium, 30 days for
[14:53.870 --> 14:55.230]  highs, and then seven days
[14:55.230 --> 14:56.930]  for critical findings.
[14:57.690 --> 14:59.030]  So pushing, upgrading
[14:59.030 --> 15:00.350]  these vulnerabilities to a
[15:00.350 --> 15:02.430]  ticket in Jira puts these
[15:02.430 --> 15:04.210]  already into the processes
[15:04.210 --> 15:05.850]  that our developers know.
[15:05.850 --> 15:08.350]  It doesn't add in any more
[15:08.350 --> 15:10.330]  friction. And it puts them
[15:10.330 --> 15:11.410]  into an established
[15:11.410 --> 15:13.230]  workflow that also ties to
[15:13.230 --> 15:14.910]  company-wide objectives.
[15:14.910 --> 15:15.770]  So upgrading these
[15:15.770 --> 15:16.530]  libraries just can't be
[15:16.530 --> 15:17.290]  another thing that
[15:17.290 --> 15:18.270]  developers ignore. They
[15:18.270 --> 15:18.970]  have to do it to meet
[15:18.970 --> 15:20.130]  their OKRs.
[15:21.170 --> 15:22.350]  So now that we had a new
[15:22.350 --> 15:24.410]  flow, we had a new way of
[15:24.410 --> 15:25.670]  dealing with and finding
[15:25.670 --> 15:27.210]  these vulnerable libraries.
[15:28.970 --> 15:30.970]  So our new flow is that
[15:30.970 --> 15:32.110]  Ossify would find the
[15:32.110 --> 15:33.510]  vulnerability and then
[15:33.510 --> 15:34.690]  throw it right into Jira.
[15:34.690 --> 15:35.650]  It would file a ticket.
[15:35.650 --> 15:37.130]  If multiple vulnerabilities
[15:37.130 --> 15:38.130]  were caused by the same
[15:38.130 --> 15:39.570]  package, we would roll
[15:39.570 --> 15:40.390]  those up into the same
[15:40.390 --> 15:42.490]  ticket. Say you have a
[15:42.490 --> 15:44.070]  couple of highs and a few
[15:44.070 --> 15:45.470]  mediums, but that all would
[15:45.470 --> 15:48.110]  be remediated by upgrading
[15:48.110 --> 15:48.230]  a single package to a
[15:48.230 --> 15:50.650]  certain version, that would
[15:50.650 --> 15:51.550]  be one ticket. So we
[15:51.550 --> 15:52.570]  weren't making multiple
[15:52.570 --> 15:53.350]  tickets for multiple
[15:53.350 --> 15:54.590]  vulnerabilities per se.
[15:54.590 --> 15:56.030]  Just if you upgrade this
[15:56.030 --> 15:56.950]  library, these are all
[15:56.950 --> 16:00.570]  soft. Now we can talk
[16:00.570 --> 16:01.850]  about our triage process a
[16:01.850 --> 16:03.350]  little bit. So the first
[16:03.350 --> 16:05.050]  step in this would be that
[16:05.050 --> 16:06.150]  the ticket would be filed
[16:06.150 --> 16:07.370]  and the product security
[16:07.370 --> 16:09.490]  engineer on call would
[16:09.490 --> 16:11.330]  triage it. So our triage
[16:11.330 --> 16:12.650]  process is just enough so
[16:12.650 --> 16:13.690]  that we can find how
[16:13.690 --> 16:14.450]  serious the vulnerability
[16:14.450 --> 16:16.170]  is in our systems.
[16:16.170 --> 16:17.130]  We would read the proof
[16:17.130 --> 16:17.530]  of concept from the
[16:17.530 --> 16:19.430]  database and then check
[16:19.430 --> 16:20.410]  our source code to see if
[16:20.410 --> 16:21.610]  we are in fact using those
[16:21.610 --> 16:22.710]  vulnerable functions.
[16:22.710 --> 16:23.810]  Because if we weren't,
[16:23.810 --> 16:24.790]  then it wouldn't be a
[16:24.790 --> 16:25.710]  critical vulnerability for
[16:25.710 --> 16:27.510]  us. It would be a low,
[16:27.510 --> 16:32.470]  or even maybe a non-fix.
[16:32.470 --> 16:35.090]  If we are using that, we
[16:35.090 --> 16:36.450]  might try to reproduce
[16:36.450 --> 16:37.850]  the attack. Or if it's
[16:37.850 --> 16:39.170]  used in multiple places,
[16:39.170 --> 16:40.210]  we're just going to
[16:40.210 --> 16:41.550]  determine that it's
[16:41.550 --> 16:43.410]  enough to determine the
[16:43.410 --> 16:44.790]  severity for fixing. If
[16:44.790 --> 16:46.170]  it's used in hundreds of
[16:46.170 --> 16:46.970]  places, we can't test
[16:46.970 --> 16:49.070]  every single one. So we
[16:49.070 --> 16:50.690]  would then recommend it
[16:50.690 --> 16:51.990]  be a medium or high
[16:51.990 --> 16:54.130]  depending on what we
[16:54.130 --> 16:55.730]  thought. And we're also
[16:55.730 --> 16:56.770]  going to review the fix
[16:56.770 --> 16:58.670]  that's in the upgraded
[16:58.670 --> 17:00.090]  version, just to see if
[17:00.090 --> 17:00.910]  there's anything more we
[17:00.910 --> 17:01.650]  need to consider when
[17:01.650 --> 17:02.830]  implementing the update.
[17:03.330 --> 17:04.310]  All of this is just to
[17:04.310 --> 17:05.830]  either confirm the
[17:05.830 --> 17:07.130]  severity based on the
[17:07.130 --> 17:09.230]  CVSS score, or adjust it
[17:09.230 --> 17:10.110]  as needed.
[17:12.570 --> 17:14.930]  From there, you know, we
[17:16.970 --> 17:18.390]  would find the
[17:19.370 --> 17:21.070]  responsible team. So
[17:21.070 --> 17:22.870]  after determining whether
[17:22.870 --> 17:23.510]  or not we get
[17:23.510 --> 17:25.190]  exploited, we need to
[17:25.190 --> 17:26.270]  find the responsible team
[17:27.070 --> 17:28.810]  to fix that. That team is
[17:28.810 --> 17:29.290]  then going to be
[17:29.290 --> 17:30.430]  responsible for mediation
[17:31.650 --> 17:33.510]  in the SLA time that
[17:33.510 --> 17:34.790]  we've already determined
[17:34.790 --> 17:36.470]  based on the severity.
[17:37.350 --> 17:38.490]  Lastly, once the package
[17:38.490 --> 17:39.950]  is updated, Ossify is
[17:39.950 --> 17:41.170]  going to go in and
[17:41.170 --> 17:42.250]  automatically see that
[17:42.250 --> 17:43.070]  that version is no longer
[17:43.070 --> 17:44.710]  used and will remove it
[17:44.710 --> 17:45.890]  from our list. And
[17:45.890 --> 17:46.970]  therefore, the
[17:46.970 --> 17:47.770]  vulnerability is
[17:47.770 --> 17:52.220]  remediated. Now that we
[17:52.220 --> 17:53.600]  have a working process,
[17:53.600 --> 17:55.100]  and that fits into what
[17:55.100 --> 17:55.920]  we're already doing at
[17:55.920 --> 17:57.280]  Slack, there's still a
[17:57.280 --> 17:58.260]  lot of future work to be
[17:58.260 --> 18:00.660]  done. Right now, we're
[18:00.660 --> 18:01.480]  working on getting a
[18:01.480 --> 18:03.120]  clean state for vulnerable
[18:03.120 --> 18:05.220]  packages, and working
[18:05.220 --> 18:06.400]  down our backlog to
[18:06.400 --> 18:07.280]  determine if the packages
[18:07.280 --> 18:08.460]  are currently being...
[18:08.460 --> 18:09.420]  that are currently being
[18:09.420 --> 18:10.240]  flagged are vulnerable
[18:10.240 --> 18:13.420]  and get them fixed.
[18:13.420 --> 18:14.620]  Next, this process, as
[18:14.620 --> 18:15.720]  you can tell, involves a
[18:16.160 --> 18:18.520]  manual work still. There's
[18:18.620 --> 18:19.320]  a lot of manual work for
[18:19.320 --> 18:20.020]  the product security
[18:20.020 --> 18:21.300]  engineer to find out who
[18:21.300 --> 18:23.420]  owns the vulnerable
[18:23.420 --> 18:24.760]  dependency and also
[18:25.300 --> 18:27.000]  seeing if it's exploitable.
[18:27.120 --> 18:27.980]  We would like to figure
[18:27.980 --> 18:29.160]  out a way to automatically
[18:29.160 --> 18:31.260]  figure that out instead
[18:31.260 --> 18:32.200]  of having that be a
[18:32.200 --> 18:33.240]  manual process. That's
[18:33.240 --> 18:34.820]  just going to be looking
[18:34.820 --> 18:36.820]  into how we can track
[18:37.860 --> 18:39.020]  when new libraries are
[18:39.020 --> 18:41.080]  added. And thirdly,
[18:41.080 --> 18:42.500]  we'd love to integrate
[18:42.500 --> 18:44.820]  this process into our
[18:44.820 --> 18:45.700]  CI pipeline. So
[18:45.700 --> 18:46.440]  packages that are
[18:46.440 --> 18:48.700]  vulnerable aren't uploaded
[18:48.700 --> 18:51.460]  to our code base when
[18:51.460 --> 18:52.520]  they're committed. So
[18:52.520 --> 18:54.940]  on commit, it would run
[18:54.940 --> 18:56.060]  the scanner, look for any
[18:56.060 --> 18:58.300]  open vulnerabilities, and
[18:58.300 --> 19:00.080]  then either block that or
[19:00.080 --> 19:00.860]  let it through if there
[19:00.860 --> 19:04.100]  aren't any. Thank you so
[19:04.100 --> 19:05.960]  much. Special thanks to
[19:05.960 --> 19:07.640]  Matt, who did ossify
[19:07.640 --> 19:08.820]  development for us, and
[19:08.820 --> 19:09.980]  Nikki and Oliver for a
[19:09.980 --> 19:10.880]  previous version of this
[19:10.880 --> 19:12.200]  talk. If you have any
[19:12.200 --> 19:12.780]  questions, we'll be in
[19:12.780 --> 19:14.180]  the Discord chat to answer
[19:14.180 --> 19:15.180]  them. Thank you.
